WORLD INTELLECTUAL PROPERTY ORGANIZATION 
International Bureau 




PCT 

INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(51) International Patent Classification 6 
A61B 5/05 



Al 



(11) International Publication Number: 
(43) International Publication Date: 



WO 99/16352 

8 April 1999 (08.04.99) 



(21) International Application Number: PCT/US98/19124 

(22) International Filing Date: 16 September 1998 (16.09.98) 



(30) Priority Data: 

08/939,662 



29 September 1997 (29.09.97) US 



(71) Applicant (for all designated States except US): MEDSIM 
ADVANCED RADIOLOGY MEDICAL SIMULATION 
LTD. [11VIL]; P.O. Box 62, 44100 Kfar Sava (IL). 

(71) Applicant (for TJ only): FRIEDMAN, Mark, M. [US/IL]; 1 

Alharizi, 43406 Raanana (IL). 

(72) Inventors; and 

(75) Inventors/Applicants (for US only): NADLER, Sima [IUIL]; 
5 Barkai, 44864 Kochav Yair (IL). AIGER, Dror [IIVIL]; 
46B Derech Lev Hasharon, 42823 Zoran (IL). COHEN-OR, 
Daniel [IL/1L]; 3 Elazar, 45242 Mod Hasharon (IL). 

(74) Common Representative: FRIEDMAN, Mark, M.; c/o Casto- 
rina, Anthony, Suite 207, 2001 Jefferson Davis Highway, 
Arlington, VA 22202 (US). 



(81) Designated States: AL, AM, AT, AU, AZ, BA, BB, BG, BR, 
BY, CA, CH, CN, CU, CZ, DE, DK, EE, ES, FI, GB, GE, 
GH, GM, HR, HU, ID, IL, IS, JP, KE, KG, KP, KR, KZ, 
LC, LK, LR, LS, LT, LU, LV, MD, MG, MK, MN, MW, 
MX, NO, NZ, PL, PT, RO, RU, SD, SE, SG, SI, SK, SL, 
TJ, TM, TR, TT, UA, UG, US, UZ, VN, YU, ZW, ARIPO 
patent (GH, GM, KE, LS, MW, SD, SZ, UG, ZW), Eurasian 
patent (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), European 
patent (AT, BE, CH, CY, DE, DK, ES, FI, FR, GB, GR, 
IE, IT, LU, MC, NL, PT, SE), OAPI patent (BF, BJ, CF, 
CG, CI, CM, GA, GN, GW, ML, MR, NE, SN, TO, TG). 



Published 

With international search report. 

Before the expiration of the time limit for amending the 
claims and to be republished in the event of the receipt of 
amendments. 



(54) Title: INTERVENTIONAL RADIOLOGY GUIDANCE SYSTEM 



(57) Abstract 

A system and method for guiding interventional radiological procedures. A constantly updated 3D ultrasound data volume of the area 
of the body in which the procedure is being performed is obtained. 2D ultrasound images in multiple desired orientations are generated from 
the 3D volume and displayed. Software-based image processing of the 3D volume is performed to identify, graphically mark, and track 
the movements, on the 2D ultrasound images, of the surgical instrument being used, the target of the procedure and the structures which it 
is desired to avoid. Visual and audible signals are generated to aid the operator in optimizing the 3D orientation of the surgical instrument 
relative to the target, to inform the operator when the target has been reached, and to warn the operator when the surgical instrument is 
approaching a structure which it is desired to avoid. 



FOR THE PURPOSES OF INFORMATION ONLY 
Codes used to identify States party to the PCT on the front pages of pamphlets publishing international applications under the PCT. 



AL 


Albania 


ES 


Spain 


LS 


Lesotho 


SI 


Slovenia 


AM 


Armenia 


FI . 


Finland 


LT 


Lithuania 


SK 


Slovakia 


AT 


Austria 


FR 


France 


LU 


Luxembourg 


SN 


Senegal 


AU 


Australia 


GA 


Gabon 


LV 


Latvia 


SZ 


Swaziland 


AZ 


Azerbaijan 


GB 


United Kingdom 


MC 


Monaco 


TD 


Chad 


BA 


Bosnia and Herzegovina 


GE 


Georgia 


MD 


Republic of Moldova 


TG 


Togo 


BB 


Barbados 


GH 


Ghana 


MG 


Madagascar 


TJ 


Tajikistan 


BE 


Belgium 


GN 


Guinea 


MK 


The former Yugoslav 


TM 


Turkmenistan 


BF 


Burkina Faso 


GR 


Greece 




Republic of Macedonia 


TR 


Turkey 


BG 


Bulgaria 


HU 


Hungary 


ML 


Mali 


TT 


Trinidad and Tobago 


BJ 


Benin 


IE 


Ireland 


MN 


Mongolia 


UA 


Ukraine 


BK 


Brazil 


IL 


Israel 


MR 


Mauritania 


UG 


Uganda 


BY 


Belarus 


IS 


Iceland 


MW 


Malawi 


US 


United States of America 


CA 


Canada 


IT 


Italy 


MX 


Mexico 


UZ 


Uzbekistan 


CF 


Central African Republic 


JP 


Japan 


NE 


Niger 


VN 


Viet Nam 


CG 


Congo 


KE 


Kenya 


NL 


Netherlands 


YU 


Yugoslavia 


CH 


Switzerland 


KG 


Kyrgyzstan 


NO 


Norway 


ZVV 


Zimbabwe 


CI 


Cdte d'lvoire 


KP 


Democratic People's 


NZ 


New Zealand 






CM 


Cameroon 




Republic of Korea 


PL 


Poland 






CN 


China 


KR 


Republic of Korea 


PT 


Portugal 






cu 


Cuba 


KZ 


Kazakstan 


RO 


Romania 






cz 


Czech Republic 


LC 


Saint Lucia 


RU 


Russian Federation 






DE 


Germany 


LI 


Liechtenstein 


SD 


Sudan 






DK 


Denmark 


LK 


Sri Lanka 


SE 


Sweden 






EE 


Estonia 


LR 


Liberia 


SG 


Singapore 







WO ^9/16352 PCT/US98/19124 

INTERVENTIONAL RADIOLOGY GUIDANCE SYSTEM 

FIELD AND BACKGROUND OF THE INVENTION 

The present invention relates to medical imaging technology, and, in 
particular, it concerns an interventional radiology guidance system and method. 
5 It is known that needles, biopsy instruments and other surgical 

instruments are used in many different kinds of interventional, radiologically 
guided, medical procedures. For example, tissue biopsy and fine needle 
aspiration are two methods often used to determine the nature of a tissue within 
the human body, one or both of these procedures usually being performed as a 
10 first step in determining the nature of a potentially malignant growth. Fluid 
drainage and tissue ablation are other procedures in which a needle is used; for 
example, when draining fluid from around the lungs, heart, or joints, or when 
eradicating a growth in the liver by injecting alcohol into the growth. When 
performing such procedures, radiological guidance is required so as to 
15 successfully navigate the instrument to the desired organ or tissue (hereinafter 
referred to as the target), while avoiding other organs, tissues or the like 
(hereinafter referred to as "avoidance targets"). Radiological guidance is also 
used during intraoperative procedures. 

All of these procedures are currently performed using one of several 
20 different radiological imaging modalities: two dimensional (2D) ultrasound, 
computerized tomography (CT), magnetic resonance imaging (MRI), or 
magnetic resonance therapeutics (MRT). Of these, 2D ultrasound is the most 
widely used guidance modality, due to its widespread availability, low cost, 
lack of radiation exposure, and ability to generate images in real time. 
25 2D ultrasound, however, suffers from several limitations: 

1. It is often difficult for the operator to see the instrument in the 
ultrasound image, as the acoustic contrast between the instrument 
and the surrounding tissue may be weak or variable. 
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2. Usually the instrument, such as a needle, cannot be perfectly 
oriented within and parallel to the plane of the 2D ultrasound 
image, because the physical presence of the ultrasound transducer 
of necessity displaces the surgical instrument, and vice-versa. The 

5 surgical instrument thus often passes through the plane of the 

ultrasound image at an angle. As such, it is difficult for the operator 
to be sure that the 2D ultrasound image of the end of the instrument 
truly represents the tip of the instrument, rather than merely the 
shaft of the instrument where it leaves the plane of the ultrasound 

10 image. This difficulty in identifying the tip of the instrument 

hinders the operator in appropriately orienting the instrument 
relative to the targets or avoidance targets. For similar reasons, 
when viewing a single 2D image, the operator cannot know with 
certainty that the instrument tip has entered the target, or has 

1 5 avoided an avoidance target. 

3. It may be difficult for the operator to orient the ultrasound probe 
such that both the instrument and the target (or avoidance target) 
are visualized in a single 2D imaging plane. 

4. Even if the operator can see both the instrument and the target (or 
20 avoidance target) in a single 2D image, it is difficult to determine if 

they are on the same 3 dimensional (3D) plane, both because of the 
inherent difficulty in extrapolating 3D conclusions from a single 
2D image, and because the 2D ultrasound image is derived from an 
ultrasound beam which itself is of a certain thickness, such 
25 thickness not being depicted in the viewed 2D image. As such, it is 

difficult for the operator to determine the appropriate 3D angle at 
which to insert and advance the instrument, so that the tip will enter 
the target, or avoid the avoidance target. 
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5. It is difficult for the operator to track the target (or the avoidance 
target) by eye, as it often moves along with movement of the 
instrument. 

6. If the instrument tip does enter the target, this fact cannot be 
5 graphically and definitively documented for future reference. 

As CT provides better image resolution than does 2D ultrasound, it is 
often preferred as an imaging modality for radiological guidance of 
interventional procedures. CT, however, still suffers from the same 3D 
orientational drawbacks mentioned above with regard to 2D ultrasound. In 

10 addition, CT entails exposure to radiation and does not readily provide real time 
imaging. MRI and MRT are rarely used due to their prohibitive expensive and 
lack of availability. 

An interventional radiological guidance system using mechanical, 
externally mounted, 6-degree-of-freedom (6 DOF) receivers to track the spatial 

15 location of sensors mounted on the external portion of the surgical instrument, 
and then use this spatial location data to display a graphic, on an ultrasound 
screen, depicting the 3D spatial location of the surgical instrument within the 
body, has been described by King (US patent # 5,608,849; issued March 4, 
1997). This system, however, does not identify and track the targets and 

20 avoidance targets of the interventional medical procedure. Furthermore, several 
deficiencies in the use of mechanical sensors to identify and track a surgical 
instrument within a body make this system innacurate: 

1. Sensor-based spatial location data needs to be accurately aligned 
with external and internal reference coordinates. This process of 

25 mechanical alignment is a significant source of data inaccuracy. 

2. Once the spatial location data for the surgical instrument has been 
determined, a graphic, depicting the instrument, needs to be 
accurately superimposed on the appropriate pixels in the 2D 
ultrasound image being viewed by the operator. As this process 

30 cannot be 100% precise, a further source of inaccuracy is introduced. 
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3. The instrument's resiliency, that is, the degree to which its shape 
may bend or become deformed once inserted into a body, cannot be 
appreciated by externally mounted sensors. As such, this method 
results in the instrument being depicted as if it were completely 
5 . rigid, an often invalid assumption. 

Other mechanically-based tracking systems would suffer from the above 
deficiencies, as well additional sources of inaccuracy specific to the mechanical 
technology being used. For example, the influence of external magnetic fields 
(if a magnetic tracking system is used), or interference with line of sight (if an 
10 optical system is used). 

There is therefore a need for an accurate, ultrasound based, radiological 
guidance system which aids the operator performing an interventional medical 
procedure to identify the instrument, the target, and the avoidance targets, to 
appropriately orient the instrument in three dimensions, and to confirm and 
1 5 document successful placement of the instrument relative to the target. 



SUMMARY OF THE INVENTION 

The present invention is a software based method and system for 
graphically marking the targets and avoidance targets of an interventional 
procedure, for tracking their movements and the movements of the instrument 

20 being used to perform the procedure, and for depicting desired orientational and 
spatial relationships between them. As opposed to mechanical tracking systems, 
such as those utilizing sensors (in which the spatial coordinates of the structure 
being tracked are defined in part by a mechanical process located externally to 
the body of the patient), the software based tracking system of the present 

25 invention defines the spatial coordinates of the structure being tracked with 
reference only to the software generated image of the patients body, and 
without reference to external mechanical processes. As such, this system can be 
said to be "self-referential" rather than "externally-referential". Hereinafter, 
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therefore, the phrase "self-referential" means "not defined by or involving the 
use of a mechanical sensor of any sort, be it electrical, optical, magnetic or the 
like". 

According to the teachings of the present invention there is provided an 
5 interventional radiological guidance system, including a mechanism for 
obtaining a plurality of chronologically sequential images of a body; an image 
identification software module operable to designate at least one self-referential 
spatial location, defined by a first set of spatial coordinates, within at least one 
of the chronologically sequential images, the desired self-referential spatial 

10 location corresponding to at least one structure in the body; an image tracking 
software module operable to self-referentially locate a second set of spatial 
coordinates in a chronologically subsequent image of the body, the second set 
of spatial coordinates corresponding to the first set of spatial coordinates in a 
chronologically precedent image of the body, such that the second set of spatial 

15 coordinates correspond to the at least one structure; and a graphic aids 
generator software module operable to generate at least one signal marking a 
location of the at least one structure in the chronologically sequential images, 
and operable to describe at least one relationship between a plurality of the at 
least one structures in the chronologically sequential images. The interventional 

20 radiological guidance system may further include a display unit operable to 
display the signal and the relationship, wherein the signal is displayed as a 
visual graphic. The interventional radiological guidance system may also 
further include a sound production unit operable to produce an audible signal 
corresponding to the relationship. The mechanism for obtaining chronologically 

25 sequential images of a body, which may be a human body, may include a three- 
dimensional ultrasound machine, a two-dimensional ultrasound machine, a 
computerized tomography scanner, a magnetic resonance imaging scanner, 
and/or a magnetic resonance therapeutics scanner. The at least one self- 
referential spatial location may be manually designated, and/or may be 

30 designated by an image processing software algorithm, which may be a motion 
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tracking software algorithm. The structure to which the self-referential spatial 
location corresponds may include a medical instrument, such as a surgical 
instrument being used in an interventional radiological procedure, and/or an 
organic structure, such as a target of a radiologically guided interventional 
5 medical procedure and/or a structure which it is desired to avoid in a 
radiologically guided interventional medical procedure. The relationship 
described by the graphic aids generator software module may include a desired 
orientation of one of the structures relative to a second of the structures, a 
desired minimum distance between one of the structures and a second of the 
10 structures, and/or a desired maximum distance between one of the structures 
and a second of the structures. This relationship may be displayed as a visual 
graphic. 

There is also provided an interventional radiological guidance method, 
including the steps of obtaining a plurality of chronologically sequential images 

15 of a body; designating at least one self-referential spatial location, defined by a 
first set of self-referential spatial coordinates within at least one of the 
chronologically sequential images, the desired self-referential spatial location 
corresponding to at least one structure in the body; self-referentially locating a 
second set of spatial coordinates in a chronologically subsequent image of the 

20 body, the second set of spatial coordinates corresponding to the first set of 
spatial coordinates in a chronologically precedent image of the body, such that 
the second set of spatial coordinates correspond to the at least one structure; 
and generating at least one signal marking a location of the at least one 
structure in the chronologically sequential images. The method may further 

25 include the step of displaying the signal as a visual graphic on a display. The 
images may be obtained using a three-dimensional ultrasound machine, a two- 
dimensional ultrasound machine, a computerized tomography scanner, a 
magnetic resonance imaging scanner, and/or a magnetic resonance therapeutics 
scanner. The body being imaged may be a human body. The self-referential 

30 spatial location may be designated by an image processing software algorithm, 
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which may be a motion tracking software algorithm, and/or may be manually 
designated. The structure to which the self-referential spatial location 
corresponds may be a medical instrument, such as a surgical instrument being 
used in an interventional radiological procedure, and/or an organic structure, 
5 such as a target of a radiologically guided interventional medical procedure, or 
a structure which it is desired to avoid in a radiologically guided interventional 
medical procedure. The method may also further include the steps of describing 
at least one relationship between a plurality of the structures in at least one of 
the chronologically sequential images, and displaying the relationship as a 

10 visual graphic on a display. The relationship may include a desired orientation 
of one of the structures relative to a second of the structures, a desired 
minimum distance between one of the structures and a second of the structures, 
and/or a desired maximum distance between one of the structures and a second 
of the structures. The method may further include the step of producing an 

1 5 audible signal corresponding to the relationship. 

The interventional radiology guidance system (IRGS) is thus based on 
3D ultrasound technology, real-time imaging software, and software based 
tracking algorithms. As opposed to current radiological guidance systems, the 
interventional radiology guidance system uses image processing software 

20 alone, without employing mechanical sensors, to identify and track the surgical 
instrument. 

In the preferred embodiment, the general functioning of the system is as 
follows: 

A standard 3D ultrasound machine is used to continuously acquire 3D 
25 ultrasound volumes of the area being imaged. Based on this data, real time 2D 
ultrasound images are depicted on the display of the 3D ultrasound machine, 
for viewing by the operator. By extracting data from the 3D volumes, the 
system is able to provide the operator with an unlimited number of image 
orientations. For example, the ultrasound data can be viewed from the point of 
30 view of the instrument, in relation to a particular plane of the body (coronal, 
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sagittal, transverse or oblique), or any other chosen angle. As multiple images 
can be viewed side-by-side simultaneously, 2 orthogonally oriented images can 
be displayed at the same time, thus facilitating 3D conceptualization, by the 
operator, of the spatial orientations of the structures being depicted in the 2D 
5. images. By "structure" is meant both organic structures (such as the target of an 
interventional radiological procedure or a structured which it is desired to avoid 
during an interventional radiological procedure) and non-organic structures 
such as medical instruments. "Medical instruments" refers to aspiration 
needles, injection needles, biopsy instruments, scalpels, ablation instruments, 

10 biomedical devices such as stents, biomedical prostheses and the like. 

When the surgical instrument being used to perform the procedure is 
introduced into the 3D volume, an image processing software-based tracking 
system is used to identify the instrument and track its spatial location. In order 
to improve the visibility of the instrument for the operator, a graphical 

15 representation of the instrument is superimposed on the ultrasound image of the 
instrument displayed on the screen. 

The operator then manually designates the target on the image, and 
marks its center and diameter by using a standard pointing device, such as the 
electronic calipers incorporated in the imaging system. By "manually" is meant 

20 that the target is designated by the operator by hand, rather than automatically 
by means of a software algorithm. A circle, or other graphical symbol, appears 
on or around the target in all subsequent ultrasound images, making the target 
easier to see. The system then tracks the position of the target as the soft 
tissues, transducer, and instrument move. If the target moves to the extent that it 

25 is no longer within the 3D volume, upon relocation the operator is required to 
re-mark it. If the instrument enters the defined radius of the target, the operator 
is informed that the instrument is located at or within the target by means of 
graphic and/or audible signals; for example, the structure's graphical symbol 
may change color and flash, a beep may be heard, and/or a textual message may 
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appear on the screen. A hard copy video or print picture documenting that the 
instrument has reached the target can be generated. 

In a similar manner, other structures in the body can be marked and 
tracked, so as to enable the operator to avoid them during the course of the 
5 procedure. In this circumstance, once the operator has designated the areas to 
be avoided, a graphical symbol appears on or around the structures (making 
them easier to see), and the system tracks their positions. If the instrument 
approaches to within a defined distance from the structure (either a default 
tolerance, or a distance defined by the operator) a warning is provided, for 

10 example, the structure's graphical symbol may change color and flash, a beep 
may be heard, and/or a textual message may appear on the screen. 

So as to facilitate optimization of the trajectory of the instrument with 
regard to the target, a graphical aid is displayed on the ultrasound screen. By 
reorienting the instrument in accordance with the orientation indicated by the 

15 graphic aid, the operator is able to more rapidly and accurately complete the 
procedure. 

In an additional embodiment, 6 DOF physical sensors are mounted on an 
instrument holder, and can be used to determine the spatial location of the 
instrument when the instrument is outside of the 3D volume. A graphical 

20 representation of the instrument, indicating its location and orientation, can be 
depicted alongside the ultrasound image, so as to enable the operator to 
optimize the orientation and trajectory of the instrument before it enters the 
body and the 3D ultrasound volume. 

The interventional radiology guidance system therefore increases the 

25 accuracy of interventional procedures by providing a clear image of the 
instrument in all planes desired by the physician, and by providing graphical 
(and other) aids which aid the physician in placing the instrument in the target, 
while avoiding critical structures in the body. When used in conjunction with 
6DOF sensors mounted on the surgical instrument, the interventional radiology 

30 guidance system can additionally provide information about the instrument's 
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location and orientation even before it has entered the 3D ultrasound image 
volume. 

With increased accuracy fewer cases will need to be repeated or 
transferred to other imaging modalities, resulting in lower cost. In addition to 
5 the advantages provided by the guidance aspects of the system, being able to 
verify in real time on intersecting, orthogonal planes that the instrument has 
successfully entered the target will greatly increase the confidence of the 
physician doing the procedure. 

In summary, this software based tracking system is more reliable than a 
10 mechanical sensor based system because: 

1. There is no need to perform alignment of the spatial location data 
with reference coordinate systems. 

2. Superimposition of graphic elements on the ultrasound image is 
100% precise because the spatial location of the graphic is defined 

15 by the voxels of the ultrasound image itself. 

3. Appropriate algorithm manipulation can be performed so as to 
evaluate the resiliency of the surgical instrument within the body in 
real time. 

4. External influences such as magnetic fields or line-of-sight 
20 interferences do not effect the system. 



BRIEF DESCRIPTION OF THE DRAWINGS 

The invention is herein described, by way of example only, with 
reference to the accompanying drawings, wherein: 
25 FIG. 1 is a block diagram of the software components of the IRGS 

FIG. 2 is a block diagram of the general flow of control of the IRGS 
FIG. 3 is a flow diagram describing the functioning of the algorithm of 
an instrument tracker 
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FIG. 4 is a flow diagram describing the functioning of a target, or 

avoidance target, tracker 
FIG. 5 is a diagram illustrating the functioning of a slice generator 
FIG. 6 shows the flow of control of a graphic aids generator 
5 FIG. 7a and FIG. 7b show examples of graphic aids generated by an 

IRGS when oriented in the long axis of a surgical instrument 
FIG. 8a and FIG. 8b show examples of graphic aids generated by an 

IRGS when oriented in the short axis of a surgical instrument. 

10 DESCRIPTION OF THE PREFERRED EMBODIMENTS 

The present invention is an interventional radiology guidance system 
(IRGS). The IRGS enables ultrasound guided procedures to be performed with 
increased speed and accuracy by tracking the location of the medical instrument 
(by which is meant surgical instruments as well as other instruments used in 

15 interventional medical procedures, as defined above), the target, and the 
avoidance targets, and providing the operator with a graphical representation of 
the instrument's trajectory with relation to the target. 

The principles and operation of an interventional radiology guidance 
system, according to the present invention, may be better understood with 

20 reference to the drawings and the accompanying description. 

The hardware component of the present invention is a 3D ultrasound 
machine, including a processing unit, a storage device, and a display unit. In an 
additional embodiment, 6 DOF physical sensors are mounted on an instrument 
holder. 

25 The 3D ultrasound machine automatically produces one volume after 

another as the operator scans the subject. The Kretz (Kretztechnik, Austria) and 
3D Ultrasound Inc (Durham, NC, USA) systems are examples of 3D ultrasound 
machines capable of supplying consecutive 3D volumes, and suitable for use in 
the IRGS. In the preferred embodiment, the processing unit and storage device 

30 are those present in the 3D ultrasound unit. In an alternative embodiment, the 
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processing unit and storage device are additional components added to the 3D 
ultrasound unit. A suitable processing unit would include 128MB of RAM and 
a 166MHz processor. The display unit is that of the 3D ultrasound machine. In 
a preferred embodiment the system is folly integrated into an existing 3D 
5 ultrasound machine. In an alternative embodiment the system is a stand-alone 
unit, which can be connected to a 3D ultrasound machine. 

The software components of the IRGS are located in the processing unit 
of the 3D ultrasound machine. 

Referring now to the drawings, Figure 1 is a block diagram of the 
10 software components of an IRGS 12, and their interrelationship with some 
hardware components of the system. The software components are: a 3D 
volume generator 1, an instrument tracker 5, a target marker 7, a target tracker 
8, an avoid marker 4, an avoid tracker 3, a slice generator 6, and a graphic aids 
generator 9. 

15 3D Volume Generator 1 provides 3D ultrasound volumes to IRGS 12. 

3D Volume Generator 1 is an external component located in the 3D ultrasound 
machine. Output from this component is a 3D volume. 

Target Marker 7 provides a user interface which enables the user to 
designate targets. For example, the calipers used to make measurements on an 

20 ultrasound screen can be used to indicate the location of a target in a frozen 
ultrasound image, by marking a circle on the image in one of the views 
displayed. Target Marker 7 then stores the center point and radius of the 
designated target. In this way a sphere representing the target in three 
dimensions is defined. 

25 Avoid marker 4 provides a user interface which enables the user to 

indicate structures to be avoided (avoidance targets). As with target marking, 
calipers can be used to indicate the location of structures in a frozen ultrasound 
image, by marking a circle on the image in one of the views displayed. Avoid 
marker 4 stores the center point and radius of the structures to be avoided. In 

30 this way, a sphere representing the structure to be avoided in three dimensions 
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is defined. The operator may mark multiple structures for avoidance, with avoid 
marker 4 being called each time a structure is marked for avoidance. 

Instrument tracker 5 is a software algorithm that locates the instrument 
within the volume provided by 3D Volume Generator 1. The output provided 
5 by this component is a list of x,y, and z coordinates describing the location of 
the instrument. 

Instrument tracker 5, Avoid marker 4, and Target Marker 7 are 
hereinafter collectively referred to as image identification software modules. 

The avoid tracker 3 component tracks structures that the operator wishes 

10 to avoid touching with the instrument (avoidance targets). It receives as input 
the position of the instrument, a 3D volume, and the details of the structures to 
be avoided. It then calculates the distance of the instrument from each of the 
structures than have been marked for avoidance. If the instrument is "too close" 
(according to a pre-defined tolerance) a message is sent to Graphics Aids 

15 Generator 9 to change the structure's graphical symbol color and cause it to 
flash. A textual warning message is also displayed. Once the instrument is 
moved away from the avoidance target, another message is sent to Graphics 
Aids Generator 9 to return the graphical symbol to its original color, stop the 
flashing and the remove the textual warning message. The algorithm used to 

20 track the structures to be avoided is identical to that used by instrument tracker 
5. 

Target tracker 8 is responsible for tracking the target in the 3D volume. 
The details, center point and radius, of the target are provided by Target Marker 
7 and the 3D volume is supplied by 3D Volume Generator 1. The output 
25 provided by this component is a set of coordinates representing the location and 
shape of the target. 

Slice generator 6 generates specific 2D slices from within the 3D 
volume, according to the angles and orientation designated by the operator 
during setup of the system. 
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Graphic Aids Generator 9 is responsible for graphically displaying the 
instrument, the target, the avoidance targets, and their relationships on each 
sequential ultrasound image. It receives as input the position and dimensions of 
the target, the avoidance targets and the instrument, as well as the 2D 
5 ultrasound images generated by slice generator 6. If the distance of the 
instrument from an avoidance target is "too close" (less than a pre-defined 
tolerance) then the avoidance targets graphical symbol color is changed (to red 
for example) and the symbol begins to flash. A textual warning is also 
displayed. Once the instrument is moved away from the structure, its graphical 
10 symbol returns to its original color, stops flashing, and the textual message 
disappears. When the instrument is inside the target, the target symbol's color 
changes (to green for example) and the target symbol begins to flash - showing 
that the user has succeeded in navigating the instrument to the desired location. 
If a 6DOF sensor system is used then it is possible for IRGS 12 to 
15 receive an instrument location which is outside of the 3D volume. In such a 
case, graphics aids generator 9 draws the instrument outside of the 2D 
ultrasound image, in a position representative of the instrument's location with 
respect to the 3D volume. 

A shared memory unit 2 stores 3D volume data generated by 3D Volume 
20 Generator 1 of the 3D ultrasound system. 

A video card 10 receives output from all of the software components, 
and a display unit 11, for displaying input received from video card 10, are also 
shown. 

A block diagram of the general flow of control of IRGS 12 is depicted 
25 in Figure 2. The first step in the iterative process is to access the 3D volume 
placed in shared memory 2 by 3D volume generator 1 of the 3D ultrasound 
system. Next a check 13 is performed to see if there are any requests to mark 
either a target or an avoidance target. If there is a request to mark a target, 
target marker 7 is called. If there is a request to mark an avoidance target, 
30 avoid marker 4 is called. Thereafter, instrument tracker 5 is called, in order to 
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identify and find the location of the instrument. If there is a marked target 14, 
Target Tracker 8 is then called in order to find the target's location in the 
current 3D volume. Following that, avoid tracker 3 is called for each of the 
marked avoidance targets (if any) 15. Then, slice generator 6 is called to 
5 produce the 2D slices corresponding to each of the views. Finally, graphic aids 
generator 9, which draws the appropriate graphic aids onto the 2D slices, is 
called. Upon completion, the slices are sent to video card 10 for display on 
display unit 11, and the flow is repeated. 

Figure 3 is a flow diagram describing the functioning of the algorithm of 

10 instrument tracker 5. The system works in an iterative fashion, identifying and 
locating the instrument in each consecutive 3D volume it receives. An N depth 
FIFO queue is used to store sub-volumes. The first step is to read the 3D 
volume T n from shared memory 2 . The second step is to define a sub-volume 
of T n , T s , surrounding the instrument. If this is one of the first seven volumes 

15 received 16, the sub-volume is determined by a predefined location 17 (usually 
the surface of the body) or, in an alternative embodiment, by information 
received from a 6DOF sensor system. Otherwise the sub-volume is determined 
18 by the location of the instrument in volume T n .|. The sub- volume T s is 
placed 19 in the FIFO and T S . N is removed 20 from the FIFO (assuming that 

20 this is the 9 lh iteration or more). T s is then registered 21 to T s _, using a standard 
registration algorithm such as that described by Barber (Barber D. C, 
Registration of low resolution medical images. Phys. Med. BioL, 27(3), pp. 87- 
96, 1992), which is incorporated herein by reference. Following this, the 
variances are calculated 22 on all of the voxels of all of the sub-volumes in the 

25 FIFO and normalized to grey level values. By calculating the variances of each 
voxel as it changes from frame to frame the instrument is identified by 
identifying its motion. (Because the instrument is typically difficult to see in 
ultrasound images, standard edge detection or feature detection alone is often 
inadequate to identify the instrument, whereas the above described motion 
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tracking software algorithm is more consistently successful at identifying the 
instrument.) 

Next, edge detection is performed 23 on the sub-volume T s . A standard 
edge detection algorithm such as that described by Canny, and which is 

5. incorporated herein by reference (J.F. Canny. A computational approach to 
edge detection. IEEE Trans. On Pattern Analysis and Machine Intelligence, 
8(6):679-698, November 1986), is used. Using the edges produced by the edge 
detection algorithm, a 3D Hough Transform 24 (R.O. Duda and P.E. Hart, 
"Use of the Hough Transform To Detect Lines and Curves in Pictures", 

10 Communications of the ACM, vol. 15, no. 1, pp. 11-15, 1972), which is 
incorporated herein by reference, is performed. The result is a list of lines (L s ). 
If this is the first iteration 25, then L s is stored 29 as L s _, and the iteration is 
completed. If this is not the first iteration 25, then L s and L s _, are matched 26 
into pairs of lines. These pairs of lines are then used to calculate 27 a flow list, 

15 as described by Adiv, incorporated herein by reference (G. Adiv, Determining 
three-dimensional motion and structure from optical flow generated by several 
moving objects. IEEE Trans, on Pattern Analysis and Machine Intelligence, 
7(4):384-401, 1985), the output of the flow list being a list of vectors. On this 
vector list segmentation is performed 28, as described by Adiv, 1985 and 

20 incorporated herein by reference. The purpose of the segmentation is to 
identify which of the lines is most likely to represent the instrument. The 
output of the segmentation is the identified instrument. The x, y, and z points 
comprising the instrument are then sent 50 to Graphic Aids Generator 9. The 
final step is to store 29 L s as L s _, and then the next iteration begins. 

25 If a 6DOF sensor system, mounted on the instrument (or its holder) and 

the transducer, is being used, then instrument tracker 5 first calls the 6DOF 
sensor system to determine if the instrument is within the 3D volume. If it is 
not within the 3D volume, position data supplied by the sensor to instrument 
tracker 5 is output directly by instrument tracker 5, without using the algorithm 

30 described above to determine the output of instrument tracker 5. If the 
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instrument is within the 3D volume, the above described algorithm is 
implemented. The advantage of using the 6DOF sensors is that when the 
instrument is inside the 3D volume, the initial sub-volume (T s ) searched for the 
instrument is defined by position data received from the sensor and is thus 
5 much smaller. This improves the performance of the instrument tracking 
algorithm. 

Figure 4 is a flow diagram describing the functioning of the algorithm 
for target tracker 8. The system works in an iterative fashion, locating the target 
in each consecutive 3D volume it receives. Two 3D volumes are required to 

10 find the target. On the first iteration, the previous volume, T n .,, is the frozen 
volume used by target marker 7, and prev_target is that supplied by target 
marker 7. If it is not the first iteration then T^ and prev target are from the 
previous iteration. T n , the current volume, is read from shared memory 2. 
Then, using the two volumes and the details of the previous target, registration 

15 30 is performed as described by Barber, 1992 and incorporated herein by 
reference. Registration identifies an area in the volume corresponding to the 
new target. The target's position, defined as its center point, is calculated 31 
and passed to Graphic Aids Generator 9. Finally, before continuing on to the 
next iteration, the target is stored 32 as prev_target and T n is stored 33 as T n . h 

20 The next iteration begins by once again reading a 3D volume from shared 
memory 2. 

Figure 5 is a diagram illustrating the functioning of slice generator 6. 
Each slice is a virtual image frame buffer defined in a world coordinate system. 
The voxels pierced by the virtual frame are sampled, mapped and displayed in 

25 their image coordinate system after the frame is clipped against the volume 
buffer. In the figure, "P" stands for "point", with PI being point #1. B is 
represents the 3D point mapped to 2D. A voxelization algorithm for planar 
polygons, as described by D. Cohen and A. Kaufman, (Scan conversion 
algorithms for linear and quadratic objects, in: A. Kaufman, Ed., Volume 

30 Visualization, IEEE Computer Society Press, Los Alamitos, CA, 1990, pages 
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280-301) and incorporated herein by reference, is used. The algorithm is 
basically an extension of the widely known 2D scan-line algorithm (J.D. Foley 
and A. van Dam and K. Feiner and F. Hughes, in Computer Graphics 
Principles and Practice, Addison- Wesley, 1990.) where at each scan-line the 
5 third dimension is also interpolated. Alternatively, the sweeping technique 
described by Cohen, Kaufman and Kong and incorporated herein by reference, 
(D. Cohen-Or, A. Kaufman and T.Y. Kong, On the Soundness of Surface 
Voxelizations, in Topological Algorithms for Digital Image Processing y T. 
Yung Kong and A. Rosenfeld (eds.), North-Holland, Amsterdam, 1995, pages 

10 181-204.) can be used. In this technique a polygon is generated by replicating 
one discrete line over the other, thus saving most of the computations involved 
in the discretization process of the plane. The output generated by slice 
generator 6 is one 2D ultrasound image for each view chosen by the operator. 
Because the 3D coordinates describing the orientation and tip of the instrument 

15 within the 3D volume are known, slice generator 6 is able to calculate the 4 
points which define a plane with an orientation within the long or short axes of 
the instrument. It is thus a feature of the IRGS that the operator can choose to 
display a 2D ultrasound plane which is consistently oriented within the long or 
short axes of the instrument, regardless of the motion of the instrument during 

20 the procedure. It will be appreciated that multiple other orientations of the 2D 
imaging plane can be chosen by the operator. In a further embodiment of the 
current invention, a 3D ultrasound image could be rendered from the 3D 
volume, for viewing by the operator, instead of individual 2D ultrasound 
images being rendering by slice generator 6. 

25 Figure 6 shows the flow of control of the graphic aids generator 9 

module. This component works iteratively. For each slice received 34 the 
following steps are performed. First the 3D representation of the instrument is 
mapped 35 to a 2D representation and the instrument is drawn 36. Then, if 
there is a target 37 the 3D representation of the target is mapped to a 2D 

30 representation. If the instrument has hit the target 39 then the target is drawn 
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40 as a flashing green object. If the target hasn't been hit then the target is 
drawn 41 without flashing. At this point a loop 42 begins in which each avoid 
object is processed. For each avoid object, the 3D representation thereof is 
mapped 43 to a corresponding 2D representation and is drawn in one of two 
5 ways. If the instrument hits the avoid object then the avoid object is drawn 44 
as a flashing red object. Otherwise the avoid object is drawn normally 46. 

Figures 7a, 7b, 8a, and 8b show examples of graphic aids generated by 
graphic aids generator module 9 to aid the operator in orienting the instrument 
relative to the target. In figures 7a and 7b the imaging slice has been chosen (by 

10 the operator) to constantly be in the long axis of the surgical instrument. As 
such, the full length of the instrument, including the tip of the instrument, is 
seen. The circle represents the target, the arrow represents the instrument, and 
the dotted line represents a projection from the target to the instrument. When 
the dotted line and the instrument form a straight line, as in figure 7b, the 

15 instrument orientation is optimal. In figure 8, the imaging plane is in the short 
axis of the surgical instrument. The light circle represents the target and the 
dark circle represents the instrument. Only when the dark circle is located 
inside the light circle will the instrument penetrate the target if inserted further, 
as is seen in Figure 8b. 

20 The above description related to image processing of an ultrasound 

image extracted from a 3D volume of ultrasound image data. It will be 
understood that by using the same software components (with the exception of 
slice generator 6, which would not be utilized), the image processing functions 
of marking and tracking of the instrument, targets and avoidance targets could 

25 be performed by IRGS 12 using 2D ultrasound image data instead of 3D data. 
In this instance, however, the operator would be unable to freely select a 
desired angle at which to generate the ultrasound image, and could not view 
more than one image simultaneously. In addition, use of 2D ultrasound data 
rather than 3D data would preclude a 3D analysis of the relationships between 

30 the viewed structures. As such, IRGS 12 would describe only the 2D 
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relationships between the target, instrument and avoidance targets. Similarly, it 
will be further understood that additional 2D or 3D digital image data sources 
(other than ultrasound) - such as CT, MRI, MRT, and the like- can be used to 
perform the above described image processing functions (marking and tracking 
5. of the instrument, targets and avoidance targets, and description of the 
relationships between them), by using the same software components, provided 
that the image data source generates chronologically sequential images 
allowing for the depiction of dynamic events. The phrase "mechanism for 
obtaining chronologically sequential images" hereinafter refers to all such 
10 image data sources. 
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WHAT IS CLAIMED IS: 

1 . An interventional radiological guidance system, comprising 

a) a mechanism for obtaining a plurality of chronologically 
sequential images of a body 

b) an image identification software module operable to designate at 
least one self-referential spatial location, defined by a first set of 
spatial coordinates, within at least one of said chronologically 
sequential images, said self-referential spatial location 
corresponding to at least one structure in said body 

c) an image tracking software module operable to self-referentially 
locate a second set of spatial coordinates in a chronologically 
subsequent image of said body, said second set of spatial 
coordinates corresponding to said first set of spatial coordinates in 
a chronologically precedent image of said body, such that said 
second set of spatial coordinates correspond to said at least one 
structure, and 

d) a graphic aids generator software module operable to generate at 
least one signal marking a location of said at least one structure in 
said chronologically sequential images, and operable to describe 
at least one relationship between a plurality of said at least one 
structures in said chronologically sequential images. 

2. The system of claim 1, wherein said mechanism is selected from the 
group consisting of three-dimensional ultrasound machines, two-dimensional 
ultrasound machines, computerized tomography scanners, magnetic resonance 
imaging scanners, and magnetic resonance therapeutics scanners. 

3. The system of claim 1, wherein said body is a human body. 
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4. The system of claim 1, wherein said at least one self-referential spatial 
location is designated by an image processing software algorithm. 

5. The system of claim 4, wherein said image processing software 
algorithm is a motion tracking software algorithm. 

6. The system of claim 1, wherein said at least one self-referential spatial 
location is manually designated. 

7. The system of claim 1, wherein said structure is selected from the group 
consisting of medical instruments and organic structures. 

8. The system of claim 7, wherein said medical instrument is a surgical 
instrument being used in an interventional radiological procedure. 

9. The system of claim 7, wherein said organic structures are selected from 
the group consisting of a target of a radiologically guided interventional 
medical procedure, and a structure which it is desired to avoid in a 
radiologically guided interventional medical procedure. 

10. The system of claim 1, further comprising 

e) a display unit operable to display said at least one signal and said 
at least one relationship, wherein said at least one signal is 
displayed as a visual graphic. 

11. The system of claim 1, wherein said at least one relationship is selected 
from the group consisting of a desired orientation of one of said at least one 
structure relative to a second of said at least one structure, a desired minimum 
distance between one of said at least one structure and a second of said at least 
one structure, and a desired maximum distance between one of said at least one 
structure and a second of said at least one structure. 
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12. The system of claim 11, wherein said relationship is displayed as a 
visual graphic. 

13. The system of claim I, further comprising 

e) a sound production unit operable to produce an audible signal 
corresponding to said at least one relationship. 

14. An interventional radiological guidance method, comprising the steps of 

a) obtaining a plurality of chronologically sequential images of a 
body 

b) designating at least one self-referential spatial location, defined by 
a first set of self-referential spatial coordinates within at least one 
of said chronologically sequential images, said self-referential 
spatial location corresponding to at least one structure in said 
body 

c) self-referentially locating a second set of spatial coordinates in a 
chronologically subsequent image of said body, said second set of 
spatial coordinates corresponding to said first set of spatial 
coordinates in a chronologically precedent image of said body, 
such that said second set of spatial coordinates correspond to said 
at least one structure, and 

d) generating at least one signal marking a location of said at least 
one structure in said chronologically sequential images. 

15. The method of claim 14, wherein said images are obtained using a 
mechanism selected from the group consisting of three-dimensional ultrasound 
machines, two-dimensional ultrasound machines, computerized tomography 
scanners, magnetic resonance imaging scanners, and magnetic resonance 
therapeutics scanners. 
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16. The method of claim 14, wherein said body is a human body. 

17. The method of claim 14, wherein said at least one self-referential spatial 
location is designated by an image processing software algorithm. 

18. The method of claim 17, wherein said image processing software 
algorithm is a motion tracking software algorithm. 

19. The method of claim 14, wherein said at least one self-referential spatial 
location is manually designated. 

20. The method of claim 14, wherein said structure is selected from the 
group consisting of medical instruments and organic structures. 

21. The method of claim 20, wherein said medical instrument is a surgical 
instrument being used in an interventional radiological procedure. 

22. The method of claim 20, wherein said organic structures are selected 
from the group consisting of a target of a radiologically guided interventional 
medical procedure, and a structure which it is desired to avoid in a 
radiologically guided interventional medical procedure. 

23. The method of claim 14, further comprising the step of 

e) displaying said at least one signal as a visual graphic on a 
display. 

24. The method of claim 23, further comprising the steps of 

f) describing at least one relationship between a plurality of said 
structures in at least one of said chronologically sequential 
images, and 
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g) displaying said relationship as a visual graphic on a display. 

25. The method of claim 24, wherein said relationship is selected from the 
group consisting of a desired orientation of one of said structures relative to a 
second of said structures, a desired minimum distance between one of said 
structures and a second of said structures, and a desired maximum distance 
between one of said structures and a second of said structures. 

26. The method of claim 24, further comprising the step of producing an 
audible signal corresponding to said relationship. 
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